Goto

Collaborating Authors

 best teacher


Merge-of-Thought Distillation

arXiv.org Artificial Intelligence

Efficient reasoning distillation for long chain-of-thought (CoT) models is increasingly constrained by the assumption of a single oracle teacher, despite the practical availability of multiple candidate teachers and growing CoT corpora. We revisit teacher selection and observe that different students have different "best teachers," and even for the same student, the best teacher can vary across datasets. Therefore, to unify multiple teachers' reasoning abilities into a student to overcome conflicts among various teachers' supervision, we propose Merge-of-Thought Distillation (MoT), a lightweight framework that alternates between teacher-specific supervised fine-tuning branches and weight-space merging of the resulting student variants. On competition math benchmarks, using only about 200 CoT samples, applying MoT to a Qwen3-14B student surpasses strong models including Deepseek-R1, Qwen3-32B, and OpenAI-O1, demonstrating substantial gains. Besides, MoT consistently outperforms the best single-teacher distillation, improves general reasoning beyond mathematics while reducing catastrophic forgetting, and shows robustness to distribution-shifted and peer-level teachers. Finally, we have demonstrated MoT possesses consensus CoT by eliminating teacher-specific inductive biases and inter-teacher conflicts while repeatedly reinforcing the learning of consensus reasoning features. These results position MoT as a simple, effective route to efficiently distilling long CoT capabilities from diverse teachers into compact students.


Why Children Are The Best Teachers For Artificial Intelligence

#artificialintelligence

Children are the best examples when it comes to someone learning new things on their own. As whatever they learn is unsupervised and mostly learned by themselves without expecting rewards, children are the best teachers for artificial intelligence. Louisa May Alcott said, "I am not afraid of storms for I am learning to sail my ship." It seems like in today's era, artificial intelligence (AI) is sailing in the same boat with us. However, considering the recent advancements artificial intelligence is having, it looks like, it still has many things to learn before coming close to humans. When we say close to humans, many think about mimicking human behaviour, but that not really is the case.


Adversary Is the Best Teacher: Towards Extremely Compact Neural Networks

AAAI Conferences

Why is our contribution important to the community? The recent boom in deep neural networks has resulted in Learning without any explicit supervision for a task ipso their being used for a wide variety of applications, many of facto provides interesting properties to our approach. An example which find significance when run on memory-constrained is that the learning method is domain and task independent, environments. Popular methods for neural network compression since instead of learning a given task, we learn aim to achieve a reduction in the number of parameters a way to learn that from the teacher. Hence, it should be while retaining state-of-the-art results. A seminal work well suited to classification, retrieval, clustering or any other on model compression was by Hinton et al [2] who introduced method across domains. Another interesting fact about this a technique in which a small student network learns approach is that humans learn in a similar way too - they from a large teacher network that is trained to saturation.